Image capture system with motion compensation

ABSTRACT

An example image system may include a lens that produces an image, an image sensor, an image stabilizer, and a controller. This image sensor has a first edge and an opposite second edge. The first edge is placed closer to lens such that it focuses on more distant objects. The image stabilizer provides a time-varying compensation of image motion at the image sensor. The controller operates the image capture system in a repeating cycle where the sensor exposes and reads out an image progressively from one edge to the opposite edge. The controller operates the image stabilizer to provide an image motion compensation that varies in time such that the image motion compensation is greater when exposing and reading the second edge of the sensor than when exposing and reading the first edge of the sensor.

BACKGROUND OF THE INVENTION

Defocus blur, motion blur, and noise due to short exposures at low lightlevels often limit cameras capturing images while in motion. Inaddition, cameras typically focus on a shallow range of depths.

BRIEF SUMMARY OF THE INVENTION

Aspects of the disclosure provide an image capture system. The imagecapture system includes a lens that produces an image; an image sensorhaving a first edge and a second edge opposite the first edge, the firstedge being placed closer to lens such that the lens focuses on moredistant objects; an image stabilizer configured to provide atime-varying compensation of image motion at the image sensor; and acontroller configured to (1) operate the image capture system in arepeating cycle, wherein during each cycle the controller operates thesensor to expose and read out an image progressively from one edge tothe opposite edge, and (2) operate the image stabilizer to provide animage motion compensation that varies in time such that the image motioncompensation is greater when exposing and reading the second edge of thesensor than when exposing and reading the first edge of the sensor.

In one example, the image stabilizer includes an oscillation of theimage capture system that is a rotational oscillation about an axisnormal to a direction of the image motion. In this example, therotational oscillation of the image capture system is about a verticalaxis. Also in this example, the placement of the image sensor relativeto the lens is unaffected by the oscillation of the image capturesystem.

In a further example, the controller is also configured to operate theimage stabilizer to provide an amount of image motion compensation thatcompensates for the image motion for objects at a first distance whenexposing and reading the first edge of the sensor and for objects at asecond distance when exposing and reading the second edge of the sensor,the first distance being greater than the second distance. In thisexample, the controller is also configured to operate the imagestabilizer to provide an amount of image motion compensation thatcompensates for the image motion for objects between the first andsecond distances when exposing and reading portions of the image sensorbetween the first and second edges.

In yet another example, the controller is also configured to process thecomposite image to extract textual information from objects in thecomposite image. In a further example, the image sensor is arolling-shutter image sensor with rows oriented vertically and readouttimes progressing from back to front. In another example, the imagestabilizer compensation comprises a rotational oscillation of the imagesensor about an axis normal to a direction of the image motion. In thisexample, the oscillation of the one or more sensors includes a rotationof 10 degrees or less.

Other aspects of the disclosure provide a method. The method includesoperating, using one or more controllers, an image capture system in arepeating cycle to capture a plurality of images, the image capturesystem having a lens, an image sensor, and an image stabilizer, whereinthe image sensor has a first edge and a second edge opposite the firstedge, the first edge being placed closer to lens such that it focuses onmore distant objects; during each cycle, operating, using the one ormore controllers, a sensor to expose and read out an image progressivelyfrom one edge to the opposite edge, and operating, using the one or morecontrollers, the image stabilizer to provide a time-varying image motioncompensation such that the motion compensation is greater when exposingand reading the second edge of the sensor than when exposing and readingthe first edge of the sensor.

In one example, the image stabilizer comprises an oscillation of theimage capture system that is a rotational oscillation about an axisnormal to a direction of the image motion. In this example, therotational oscillation of the image capture system is about a verticalaxis. Also in this example, the angle of the image sensor is fixedrelative to the lens.

In another example, operating the image stabilizer to provide atime-varying image motion compensation also includes compensating forthe image motion for objects at a first distance when exposing andreading the first edge of the sensor, for objects at a second distancewhen exposing and reading the second edge of the sensor, the firstdistance being greater than the second distance, and for objects betweenthe first and second distances when exposing and reading portions of theimage sensor between the first and second edges. In yet another example,the method also includes processing, by the one or more controllers, theplurality of images to extract textual information from objects in theimages. In a further example, the image sensor is a rolling-shutterimage sensor with rows oriented vertically and readout times progressingfrom back to front.

Further aspects of the disclosure provide a non-transitory,computer-readable medium on which instructions are stored, theinstructions, when executed by one or more controllers, cause the one ormore controllers to perform a method. The method includes operating animage capture system in a repeating cycle to capture a plurality ofimages, the image capture system having a lens, an image sensor, and animage stabilizer, wherein the image sensor has a first edge and a secondedge opposite the first edge, the first edge being placed closer to lenssuch that it focuses on more distant objects; during each cycle,operating a sensor to expose and read out an image progressively fromone edge to the opposite edge, and operating the image stabilizer toprovide a time-varying image motion compensation such that the motioncompensation is greater when exposing and reading the second edge of thesensor than when exposing and reading the first edge of the sensor.

In one example, operating the image stabilizer to provide a time-varyingimage motion compensation also includes compensating for the imagemotion for objects at a first distance when exposing and reading thefirst edge of the sensor, for objects at a second distance when exposingand reading the second edge of the sensor, the first distance beinggreater than the second distance, and for objects between the first andsecond distances when exposing and reading portions of the image sensorbetween the first and second edges. In another example, the method alsoincludes processing the plurality of images to extract textualinformation from objects in the images.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional diagram of an exemplary traditional image capturesystem according to aspects of the disclosure.

FIG. 2 is a functional diagram of an example image capture systemaccording to aspects of the disclosure.

FIG. 3 is a pictorial diagram of the image capture system of FIG. 2 inaccordance with aspects of the disclosure.

FIG. 4 is a functional diagram of an image capture system according toaspects of the disclosure.

FIG. 5A is a graph representing an example operation of the imagecapture system according to aspects of the disclosure.

FIG. 5B is a graph representing an example operation of the imagecapture system according to aspects of the disclosure.

FIG. 6 is a functional diagram of an example system in accordance withan exemplary embodiment.

FIG. 7 is a functional diagram of an example system in accordance withan exemplary embodiment.

FIG. 8 is a pictorial diagram of the system of FIG. 2 in accordance withaspects of the disclosure.

FIG. 9 is a flow diagram of an example method according to aspects ofthe disclosure.

DETAILED DESCRIPTION Overview

The technology relates to an image capture system that is able tocapture images having a range of different distances in focus while thevehicle is in motion. The image capture system may be mounted ormountable on a vehicle.

Typical camera systems may utilize image sensors oriented parallel to aplane of a lens. In systems in which the image sensors are parallel tothe lens, the distance at which objects are in focus is the same acrossthe image sensor. These systems may utilize camera stabilizer systems,liquid wedge prism lenses, sensor shift, or post-processing adjustmentsin order to attempt to compensate for small movements and shakes due toa user's hand movement. However, when attempting to adapt for themovements of a vehicle these techniques have well-understood limits,making them less effective at addressing quality issues such as defocusblur, motion blur, and noise due to short exposures at low light levels.When such images are blurry or noisy, this may make it difficult torecognize certain features in images, such as text on signs, when usingtechniques such as optical character recognition (OCR). As a result, inorder to capture the variety of shots required for OCR, typical camerasystems may require either multiple passes or multiple cameras.

In order to improve the quality of images captured by the image capturesystem while the vehicle is in motion, an image capture system mayutilize an image sensor installed on a tilt in relation to a lens ratherthan parallel to the lens. The image sensor in the image capture systemmay be tilted at an angle, with a first edge slightly closer to thelens, to focus best on distant objects, than a second edge opposite thefirst edge, which focuses best on near objects. The amount of tiltrequired to cover a range of object distances is approximately linear inreciprocal object distance: i=f+f²/o, where image distance i (from imagesensor to lens place) is slightly more than focal length f, by an amountproportional to the reciprocal of object distance o. As a result, thefirst edge of the image sensor may be focused on objects farther awayfrom the camera than objects on which the second edge of the imagesensor is focused, such that objects that appear in different positionsin more than one image may be in better focus in one than in another.

The image sensor may be a rolling-shutter image sensor with rowsoriented vertically, or in portrait mode, and readout times progressingfrom back to front, or vice versa. The rolling shutter reads out animage starting at one edge of the image sensor and progressing to theopposite edge, such that different portions of the image are read atslightly different times. By applying a time-varying image stabilizationrelated to vehicle velocity, the motion blur due to vehicle motion maybe compensated for objects at different distances in different portionsof the image. Image stabilization may be applied by various known means,including moving the image sensor behind the lens, rotating the wholecamera, or moving the image using a liquid-wedge prism.

The system may rotate on an axis normal to the direction in which thevehicle is traveling and normal to the direction in which the camera ispointing in order to compensate for vehicle movement; for example, avertical axis if the camera is looking to the side of a moving car. Therotation may be a torsional oscillation and may also be sinusoidal orany other type of oscillation. In other words, the system may rotateback and forth on the axis, swinging the lens in the direction of traveland then opposite the direction of travel. During oscillation of thesystem, the orientation of the image sensor relative to the lens mayremain fixed. When rotating opposite the direction of travel, the systemmay compensate for vehicle movement. This compensation may allow thesystem to capture clearer images of stationary objects that wouldotherwise be blurry due to the vehicle movement. Thus, to obtain themost images of higher quality, system may be set up to only captureimages when the system is rotating opposite the direction of vehicletravel.

The combination of the tilted image sensor and rotating system may allowthe system to compensate for varied angular velocities of multipleplanes of focus. By nature of being a short distance away from a movingvehicle, objects close to the vehicle have a higher angular velocitythan objects farther away from the vehicle. Objects at infinity havezero angular velocity. The angular velocity of an object changes in alinear fashion in relation to the reciprocal distance of the object fromthe system. When the tilted image sensor is rotated, the angularvelocity of the first edge of the image sensor may be faster than theangular velocity of the second edge. This arrangement may compensate forthe difference in angular velocity at different distances away from thevehicle and allow the system to capture portions of the image that areboth well focused and well motion compensated for objects at differentdistances in different parts of the image.

To further compensate for the difference in angular velocity, therotation of the system may be synchronized with a rolling shutterexposure and readout, and the amount of rotational velocity may beadjusted to be proportional to the vehicle velocity, such that the imagemotion on the image sensor is approximately stopped for objects that arein focus in each region of the image sensor.

The system may capture images at a high enough rate that objects ofinterest appear in several successive images, but are clearer or more infocus in one image or another depending on their distances and on theirpositions within the image. Because the tilted image sensor may providea gradient of focal distances across a single image, neighboring imagesmay capture the same object or location at different image locationshaving different best focus distances.

The captured images may be processed in order to extract textualinformation from objects in the image. Having been generated from imagesthat have a variety of focus distances and have been compensated formotion as described above, traditional machine-reading methods may beapplied.

Example Systems

FIG. 1 is a functional diagram of a traditional image capture system.Image capture system 110 may have lens 120 and an image sensor 130arranged parallel to one another. As a result, the plane of focus 140 isalso parallel to the lens.

An image capture system may utilize an image sensor installed on a tiltin relation to a lens rather than parallel to the lens in order toimprove the quality of images captured by the image capture system whilethe vehicle is in motion. As shown in FIG. 2, an image capture system210 may have a lens 220 and image sensor 230. The image sensor 230 maybe tilted at an angle α, with a first edge slightly closer to the lens,to focus best on distant objects, than a second edge opposite the firstedge, which focuses best on near objects. The amount of tilt required tocover a range of object distances is approximately linear in reciprocalobject distance: i=f+f²/o, where image distance i (from image sensor tolens place) is slightly more than focal length f, by an amountproportional to the reciprocal of object distance o.

For example, lens 220 may have a focal length 0.010 m (10 mm), with fardistance of infinity, the near edge of the image sensor 230 may beplaced 0.010 m behind the lens plane; with near object distance of 4 m,the opposite edge of the image sensor 230 may be placed 0.010025 mbehind the lens plane. This slight tilt of 25 microns (0.000025 m) fromone edge to the other corresponds to an angle of 2.5 milliradians (about0.14 degree) if the image sensor edges are 10 mm apart. As a result, thefirst edge of the image sensor may be focused on objects farther awayfrom the camera than objects on which the second edge of the imagesensor 230 is focused, such that objects that appear in differentpositions in more than one image may be in better focus in one than inanother.

The specific amount of tilt may be determined to be that which bestcovers the range of object distances to be imaged. As noted above, thedirection of tilt may be about an axis normal to the direction in whichthe vehicle is traveling, such as a vertical axis, but may be in anyother direction as well.

The image sensor 230 may be a rolling-shutter image sensor with rowsoriented vertically, or in portrait mode, and readout times progressingfrom back to front, or vice versa. The rolling shutter reads out animage starting at one edge of the image sensor and progressing to theopposite edge, such that different portions of the image are read atslightly different times. The image may also be a collection of pixelphotosensors in a single array on one chip or on a plurality of chips.Additionally or alternatively, the system may utilize a plurality ofimage sensors arranged on an angle behind the lens. An image sensor at afirst edge of the lens may be a smaller distance from the lens than animage sensor at a second edge of the lens. Each of the plurality ofimage sensors may be tilted as described above. The plurality of imagesensors may be configured to behave as a single rolling-shutter imagesensor; in other words, expose and readout images starting at one edgeof the lens and progressing to the opposite edge of the lens. Each imagesensor may be a rolling-shutter image sensor, but not necessarily.

FIG. 3 is a pictorial diagram of the image capture system of FIG. 2. Inaddition to lens 220 and one or more image sensors 230, image capturesystem 210 may have one or more controllers 310. The one or morecontrollers 310 may control operations of the image capture system. Forexample, the one or more controllers 310 may cause the image capturesystem 210 to move or capture an image. The one or more controllers mayalso control components of the image capture system 210, such as lens220 or image sensors 230, individually. In some examples, the imagecapture system may include a memory storing data and instructions thatmay be used executed to operate the system.

In addition to having a tilted image sensor, the system may beconfigured to rotate on an axis normal to the direction in which thevehicle is traveling and normal to the direction in which the camera ispointing in order to compensate for vehicle movement. The rotation maybe a torsional oscillation and may also be sinusoidal. In other words,the system may rotate back and forth on the axis, swinging the lens inthe direction of travel and then opposite the direction of travel. Asshown in FIG. 4, image capture system 210 may rotate back and forthabout axis 420, covering angular distance β.

During oscillation of the system, the orientation of the image sensorrelative to the lens may remain fixed. In other words, the image sensormay rotate together with the lens. For example, the system may rotateback and forth within 10 degrees of angular distance. The system maytherefore compensate for displacement of the system in the period oftime in which the system is rotating opposite the direction of travel.Put another way, the system may compensate for image motion across thesensor by rotating in the direction of image motion. The system may beconfigured to capture images when the system is rotating opposite thedirection of displacement (or in the direction of image motion) and bestcompensating for the displacement.

FIGS. 5A and 5B show the angular velocity and camera rotation angleversus time through a cycle of image capture for an image capture systemcalibrated to focus on object distances of 4 meters through infinity,according to example embodiments of the invention. FIG. 5A graphicallydepicts an example of the rotation of the image capture system asangular velocity over time shown in solid line 510. Dotted line 512represents the amount of rotation in radians. Line 514 represents theexposure and readout interval for an image, and line segment 516represents the distance with the best motion compensation based on thetilt of the image sensor and the angular velocity of the system. The +marks 518 represent the ideal angular velocity to compensate forvelocity of travel at 4 meter and 8 meter distances. In FIG. 5A, theimage capture rate is 8 frames per second (fps) with readout in 1/48second, and vehicle velocity at 10 m/s. Therefore, at 8 fps, the systemimages during ⅙ of a sinusoidal motion, highlighted by the bold line520, where the system rotation best compensates for distances rangingfrom 4 meters to infinity.

FIG. 5B graphically depicts another example of the rotation of the imagecapture system as angular velocity over time shown in solid line 530.Dotted line 532 represents the amount of rotation in radians. Line 534represents the exposure and readout interval for an image, and linesegment 536 represents the distance with the best motion compensationbased on the tilt of the image sensor and the angular velocity of thesystem. The + marks 538 represent the ideal angular velocity tocompensate for velocity of travel at 4 meter and 8 meter distances. InFIG. 5B, the image capture rate is 12 frames per second with readout in1/48 second, and vehicle velocity at 10 m/s. The higher frame rate isobtained by using a motion closer to a sawtooth waveform (slower rampone direction than the other, using up to fifth harmonic of the 12 Hzcycle). At a rate of 12 fps, the system images during ¼ of the cycle, ashighlighted by bold line 540.

In addition to or as an alternative to rotation, the system mayoscillate side-to-side as a translational oscillator in a plane parallelto the direction in which the vehicle is traveling. In yet anotherexample, rather than rotating or oscillating the whole system, only theimage sensor may be oscillated relative to the lens to compensate imagemotion.

The combination of the tilted image sensor and rotating system, mayallow the system to compensate for varied angular velocities of multipleplanes of focus. By nature of being a short distance away from a movingvehicle, objects close to the vehicle have a higher angular velocitythan objects farther away from the vehicle. For example, for a vehiclevelocity of 10 m/s, the angular velocity of an object at 4 meters awayis 2.5 rad/s, or 2.5 mrad/ms. Objects at infinity have zero angularvelocity. The angular velocity of an object changes in a linear fashionin relation to the reciprocal distance of the object from the system.When the tilted image sensor is rotated, the angular velocity of thefirst edge of the image sensor may be faster than the angular velocityof the second edge. In other words, the edge of the image sensor withthe near focus may have the faster angular velocity. This arrangementmay compensate for the difference in angular velocity at differentdistances away from the vehicle and allow the system to capture portionsof the image that are both well focused and well motion compensated forobjects at different distances in different parts of the image.

To further compensate for the difference in angular velocity, therotation of the system may be synchronized with a rolling shutterexposure and readout, and the amount of rotational velocity may beadjusted to be proportional to the vehicle velocity, such that the imagemotion on the image sensor is approximately stopped for objects that arein focus in each region of the image sensor.

The system may capture images at a high enough rate that objects ofinterest appear in several successive images, but are clearer in oneimage or another depending on their distances and on their positionswithin the image. In other words, objects may appear in more than oneimage and at different points in different images. For example, thesystem may capture one shot per oscillation cycle, each oscillationbeing within 10 degrees of rotation. This ⅙ cycle exposure and readoutmay correspond to a frame interval equal to about 6 times the readouttime, or ⅙ of the image sensor's maximum frame rate. For an image sensorcapable of reading in 1/48 s, a total of 8 images may be captured persecond, or an image every 1.25 meters if the vehicle is traveling at 10m/s, which may capture multiple views of an object that is more than afew meters from the vehicle (depending on the field of view of thecamera).

Because the tilted image sensor may provide a gradient of focaldistances across a single image, neighboring images may capture the sameobject or location at different image locations having different bestfocus distances. For example, at object at 8 m distance might appear in3 or more different images (depending on the field of view), and in atleast one of those the object appear near the center of image where thebest focus distance is 8 m (half way between the infinity and 4 m edgesin terms of reciprocal distance). If the motion compensation is adjustedto no rotation at the infinity edge and compensation for 10 m/s vehiclemotion for objects at 4 m distance (rotation of 2.5 radians per second)at the other edge, and varying approximately linearly between those,then it will approximately cancel the motion blur for the object at 8 mdistance near the center of the image. Similarly, other objects atdistances between 4 m and infinity will be both well focus and motioncompensated at the approximately corresponding image locations, so thatif they appear in several image they will be sharp in at least one.

As shown in FIG. 6, image capture system 210 may be incorporated intovehicle 600. While certain aspects of the disclosure are particularlyuseful in connection with specific types of vehicles, the vehicle may beany type of vehicle including, but not limited to, cars, trucks,motorcycles, buses, recreational vehicles, etc. The vehicle may have oneor more computing devices, such as computing device 610 containing oneor more processors 620, memory 630 and other components typicallypresent in general purpose computing devices. Image capture system 210may be connected to computing device 610 and mounted onto vehicle 600.

The memory 630 stores information accessible by the one or moreprocessors 620, including data 632 and instructions 634 that may beexecuted or otherwise used by the processor 620. The one or moreprocessors 620 may be any conventional processors, such as commerciallyavailable CPUs. Alternatively, the one or more processors may be adedicated device such as an ASIC or other hardware-based processor. Thememory 630 may be of any type capable of storing information accessibleby the processor, including a computing device-readable medium, or othermedium that stores data that may be read with the aid of an electronicdevice, such as a hard-drive, memory card, ROM, RAM, DVD or otheroptical disks, as well as other write-capable and read-only memories.Systems and methods may include different combinations of the foregoing,whereby different portions of the instructions and data are stored ondifferent types of media.

The data 632 may be retrieved, stored or modified by processor 620 inaccordance with the instructions 634. For instance, although the claimedsubject matter is not limited by any particular data structure, the datamay be stored in computing device registers, in a relational database asa table having a plurality of different fields and records, XMLdocuments or flat files. The data may also be formatted in any computingdevice-readable format.

The instructions 634 may be any set of instructions to be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor. For example, the instructions may be stored as computingdevice code on the computing device-readable medium. In that regard, theterms “instructions” and “programs” may be used interchangeably herein.The instructions may be stored in object code format for directprocessing by the processor, or in any other computing device languageincluding scripts or collections of independent source code modules thatare interpreted on demand or compiled in advance. Functions, methods androutines of the instructions are explained in more detail below.

Although FIG. 6 functionally illustrates the processor, memory, andother elements of computing device 610 as being within the same block,it will be understood by those of ordinary skill in the art that theprocessor, computing device, or memory may actually include multipleprocessors, computing devices, or memories that may or may not be storedwithin the same physical housing. For example, memory may be a harddrive or other storage media located in a housing different from that ofcomputing device 610. Accordingly, references to a processor orcomputing device will be understood to include references to acollection of processors or computing devices or memories that may ormay not operate in parallel.

Computing device 610 may have all of the components normally used inconnection with a computing device such as the processor and memorydescribed above as well as a user input 650 (e.g., a mouse, keyboard,touch screen and/or microphone) and various electronic displays (e.g., amonitor having a screen or any other electrical device that is operableto display information). In this example, the vehicle includes aninternal electronic display 652 as well as one or more speakers 654 toprovide information or audio visual experiences. In this regard,internal electronic display 652 may be located within a cabin of vehicle600 and may be used by computing device 610 to provide information topassengers within the vehicle 600.

Computing device 610 may also include one or more wireless networkconnections 654 to facilitate communication with other computingdevices, such as the client computing devices and server computingdevices described in detail below. The wireless network connections mayinclude short range communication protocols such as Bluetooth, Bluetoothlow energy (LE), cellular connections, as well as various configurationsand protocols including the Internet, World Wide Web, intranets, virtualprivate networks, wide area networks, local networks, private networksusing communication protocols proprietary to one or more companies,Ethernet, WiFi and HTTP, and various combinations of the foregoing.

In addition to image capture system 210, computing device 610 may alsobe in communication with one or more vehicle operation systems 660 ofvehicle 600. Vehicle operation systems 660 may include systems involvedin operations of the vehicle, such as one or more of deceleration,acceleration, steering, signaling, navigation, positioning, detection,etc. Although one or more vehicle operation systems 660 are shown asexternal to computing device 610, in actuality, the systems may also beincorporated into computing device 610.

Image capture system 210 may also receive or transfer information, suchas captured images, to and from other computing devices. FIGS. 7 and 8are pictorial and functional diagrams, respectively, of an examplesystem 700 that includes a plurality of computing devices 710, 720, 730,740 and a storage system 750 connected via a network 760. System 700also includes image capture system 210. Although only a few computingdevices are depicted for simplicity, a typical system may includesignificantly more. Additionally or alternatively, one or more vehiclessuch as vehicle 600 may be included in system 700.

As shown in FIG. 8, each of computing devices 710, 720, 730, 740 mayinclude one or more processors, memory, data and instructions. Suchprocessors, memories, data and instructions may be configured similarlyto one or more processors 620, memory 630, data 632, and instructions634 of computing device 610.

The network 760, and intervening nodes, may include variousconfigurations and protocols including short range communicationprotocols such as Bluetooth, Bluetooth LE, the Internet, World Wide Web,intranets, virtual private networks, wide area networks, local networks,private networks using communication protocols proprietary to one ormore companies, Ethernet, WiFi and HTTP, and various combinations of theforegoing. Such communication may be facilitated by any device capableof transmitting data to and from other computing devices, such as modemsand wireless interfaces.

In addition, server computing devices 710 may use network 760 totransmit and present information to a user, such as user 722, 732, 742on a display, such as displays 724, 734, 742 of computing devices 720,730, 740. In this regard, computing devices 720, 730, 740 may beconsidered client computing devices.

As shown in FIG. 8, each client computing device 720, 730, 740 may be apersonal computing device intended for use by a user 722, 732, 742, andhave all of the components normally used in connection with a personalcomputing device including a one or more processors (e.g., a centralprocessing unit (CPU)), memory (e.g., RAM and internal hard drives)storing data and instructions, a display such as displays 724, 734, 744(e.g., a monitor having a screen, a touch-screen, a projector, atelevision, or other device that is operable to display information),and user input devices 726, 736, 746 (e.g., a mouse, keyboard, touchscreen or microphone). The client computing devices may also include acamera for recording video streams, speakers, a network interfacedevice, and all of the components used for connecting these elements toone another.

In addition, the client computing devices 720 and 730 may also includecomponents 728 and 738 for determining the position and orientation ofclient computing devices. For example, these components may include aGPS receiver to determine the device's latitude, longitude and/oraltitude as well as an accelerometer, gyroscope or anotherdirection/speed detection device.

Although the client computing devices 720, 730, and 740 may eachcomprise a full-sized personal computing device, they may alternativelycomprise mobile computing devices capable of wirelessly exchanging datawith a server over a network such as the Internet. By way of exampleonly, client computing device 720 may be a mobile phone or a device suchas a wireless-enabled PDA, a tablet PC, a wearable computing device orsystem, or a netbook that is capable of obtaining information via theInternet or other networks. In another example, client computing device730 may be a wearable computing system, shown as a head-mountedcomputing system in FIG. 7. As an example the user may input informationusing a small keyboard, a keypad, microphone, using visual signals witha camera, or a touch screen.

Storage system 750 may store various types of information that may beretrieved or otherwise accessed by a server computing device, such asone or more server computing devices 710, in order to perform some orall of the features described herein. The storage system 750 may storeimages captured by image capture system 210. Information associated withimages such as location information and pose information may be storedin association with the images.

As with memory 730, storage system 750 can be of any type ofcomputerized storage capable of storing information accessible by theserver computing devices 710, such as a hard-drive, memory card, ROM,RAM, DVD, CD-ROM, write-capable, and read-only memories. In addition,storage system 750 may include a distributed storage system where datais stored on a plurality of different storage devices which may bephysically located at the same or different geographic locations.Storage system 750 may be connected to the computing devices via thenetwork 760 as shown in FIG. 7 and/or may be directly connected to orincorporated into any of the computing devices 710, 720, 730, 740, imagecapture device 210, etc.

Example Methods

FIG. 9 is an example flow diagram 900 in accordance with some of theaspects described above that may be performed by one or more controllersin the system. In this example, one or more image sensors may beconfigured at an angle behind a lens of an image capture system at block910. The image capture system may then be oscillated based on adirection of motion of the image capture system at block 920. Forexample, the image capture system may be mounted on a vehicle, in whichcase, the direction the vehicle is traveling would be the direction ofmotion of the image capture system. At block 930, the image capturesystem may expose and read images starting at a first edge of a lens inthe system and progressing to a second edge of the lens opposite thefirst edge. To expose and read images in such a way, a sensor of theimage capture system may be a rolling-shutter image sensor.

The captured images may be processed in order to extract textualinformation from objects in the image. Having been generated from imagesthat have a variety of focus distances and have been compensated formotion as described above, traditional machine-reading methods may beapplied. For example, words on signs may be read through use of OCR. Thecaptured images may be processed at the image capture device 210. Inother examples, the captured images may be sent via network 760 to oneor more computing devices 710, 720, 730, 740 and processed at the one ormore computing devices. The captured images and/or the extractedinformation may be sent from image capture system 210 or one or morecomputing devices 710, 720, 730, 740 via the network 760 to storagesystem 750. In response to a user request, a captured image or extractedinformation may be retrieved from storage system 750.

In some examples, a composite image may be generated by stitchingtogether select portions of the captured images that are most in focusand processed in order to extract information from the image. Thecomposite image may be a panoramic image. In order to select theportions of the captured images that are most in focus, portions ofdifferent images capturing a particular location may be compared. Theportion of a captured image that depicts a particular location theclearest or at a location closest to the focal distance may be selectedto be included in the composite image. The system may use LIDAR todetect the distance between the particular location and the vehicle. Theportion of a captured image in which the particular location is capturedat the image sensor portion focused on a distance matching the detecteddistance may be selected to be included in the composite image. Thegenerated composite image may then be processed to extract textualinformation from objects in the image.

The features described above allow for the capture of images that are infocus at a wide range of distances when traveling at a high velocity,such as when driving in a car. While each of a tilted image sensor,rotation or oscillation, or overlapping images provides individualadvantages over a typical image capture system as discussed above, thecombination of these may provide the best source of captured images forgenerating composite images as described above. As a result, thecomposite images may contain more information than what is typicallycaptured in an image taken from a moving vehicle. For example, thecomposite images may be used in post-processing to extract textualinformation like words on signs that would otherwise be too blurry orout-of-focus to read if captured by a typical image capture system. Inaddition, the tilted image sensor and the motion of the image sensorallows for the efficient capture of images in a single pass with onecamera where the images are captured at a variety of focus distances anddegrees of motion compensation. As such, the features disclosed abovemay allow for the use of wider aperture and slower shutter speeds,enabling the image capture system to get enough light to make cleanimages at a high velocity.

Although the invention herein has been described with reference toparticular embodiments, it is to be understood that these embodimentsare merely illustrative of the principles and applications of thepresent invention. It is therefore to be understood that numerousmodifications may be made to the illustrative embodiments and that otherarrangements may be devised without departing from the spirit and scopeof the present invention as defined by the appended claims.

1. An image capture system comprising: a lens that produces an image; animage sensor having a first edge and a second edge opposite the firstedge, the first edge being placed closer to lens such that the lensfocuses on more distant objects; an image stabilizer configured toprovide a time-varying compensation of image motion at the image sensor;and a controller configured to: operate the image capture system in arepeating cycle, wherein during each cycle the controller operates thesensor to expose and read out an image progressively from one edge tothe opposite edge, and operate the image stabilizer to provide an imagemotion compensation that varies in time such that the image motioncompensation is greater when exposing and reading the second edge of thesensor than when exposing and reading the first edge of the sensor. 2.The system of claim 1, wherein the image stabilizer comprises anoscillation of the image capture system that is a rotational oscillationabout an axis normal to a direction of the image motion.
 3. The systemof claim 2, wherein the rotational oscillation of the image capturesystem is about a vertical axis.
 4. The system of claim 1, wherein thecontroller is further configured to operate the image stabilizer toprovide an amount of image motion compensation that compensates for theimage motion for objects at a first distance when exposing and readingthe first edge of the sensor and for objects at a second distance whenexposing and reading the second edge of the sensor, the first distancebeing greater than the second distance.
 5. The system of claim 4,wherein the controller is further configured to operate the imagestabilizer to provide an amount of image motion compensation thatcompensates for the image motion for objects between the first andsecond distances when exposing and reading portions of the image sensorbetween the first and second edges.
 6. The system of claim 1, whereinthe controller is further configured to process the composite image toextract textual information from objects in the composite image.
 7. Thesystem of claim 1, wherein the image sensor is a rolling-shutter imagesensor with rows oriented vertically and readout times progressing fromback to front.
 8. The system of claim 2, wherein the placement of theimage sensor relative to the lens is unaffected by the oscillation ofthe image capture system.
 9. The system of claim 1, wherein the imagestabilizer compensation comprises a rotational oscillation of the imagesensor about an axis normal to a direction of the image motion.
 10. Thesystem of claim 9, wherein the oscillation of the one or more sensorsincludes a rotation of 10 degrees or less.
 11. A method comprising:operating, using one or more controllers, an image capture system in arepeating cycle to capture a plurality of images, the image capturesystem having a lens, an image sensor, and an image stabilizer, whereinthe image sensor has a first edge and a second edge opposite the firstedge, the first edge being placed closer to lens such that it focuses onmore distant objects; during each cycle, operating, using the one ormore controllers, a sensor to expose and read out an image progressivelyfrom one edge to the opposite edge, and operating, using the one or morecontrollers, the image stabilizer to provide a time-varying image motioncompensation such that the motion compensation is greater when exposingand reading the second edge of the sensor than when exposing and readingthe first edge of the sensor.
 12. The method of claim 11, wherein theimage stabilizer comprises an oscillation of the image capture systemthat is a rotational oscillation about an axis normal to a direction ofthe image motion.
 13. The method of claim 12, wherein the rotationaloscillation of the image capture system is about a vertical axis. 14.The method of claim 11, wherein operating the image stabilizer toprovide a time-varying image motion compensation further comprisescompensating for the image motion for objects at a first distance whenexposing and reading the first edge of the sensor, for objects at asecond distance when exposing and reading the second edge of the sensor,the first distance being greater than the second distance, and forobjects between the first and second distances when exposing and readingportions of the image sensor between the first and second edges.
 15. Themethod of claim 11, further comprising processing, by the one or morecontrollers, the plurality of images to extract textual information fromobjects in the images.
 16. The method of claim 11, wherein the imagesensor is a rolling-shutter image sensor with rows oriented verticallyand readout times progressing from back to front.
 17. The method ofclaim 12, wherein the angle of the image sensor is fixed relative to thelens.
 18. A non-transitory, computer-readable medium on whichinstructions are stored, the instructions, when executed by one or morecontrollers, cause the one or more controllers to perform a method, themethod comprising: operating an image capture system in a repeatingcycle to capture a plurality of images, the image capture system havinga lens, an image sensor, and an image stabilizer, wherein the imagesensor has a first edge and a second edge opposite the first edge, thefirst edge being placed closer to lens such that it focuses on moredistant objects; during each cycle a sensor to expose and read out animage progressively from one edge to the opposite edge, and operatingthe image stabilizer to provide a time-varying image motion compensationsuch that the motion compensation is greater when exposing and readingthe second edge of the sensor than when exposing and reading the firstedge of the sensor.
 19. The medium of claim 18, wherein operating theimage stabilizer to provide a time-varying image motion compensationfurther comprises compensating for the image motion for objects at afirst distance when exposing and reading the first edge of the sensor,for objects at a second distance when exposing and reading the secondedge of the sensor, the first distance being greater than the seconddistance, and for objects between the first and second distances whenexposing and reading portions of the image sensor between the first andsecond edges.
 20. The medium of claim 18, wherein the method furthercomprises processing the plurality of images to extract textualinformation from objects in the images.